Field of the Invention
The present invention relates generally to software processing, and more particularly, to methods and systems for improving computer software testing using test coverage data.
Description of the Related Art
Within the field of the invention, translating source code (human readable representation of a program, a computer language) into the target code in binary form (machine readable representation) is well-known. This process is typically called a compile or build process. FIG. 1 illustrates this with a build process 120 connecting between a respective source code 110 and the resulting executable program 130.
A software program should be sufficiently tested to determine if it operates error-free, before it is used for “production” purposes. Therefore, automated execution of test suites is also well-known prior art. A test suite is a pre-compiled list of test cases. These test case lists are created manually or semi-manually today. FIG. 2 illustrates an example of a test process. In a first step 210 the freshly built program (i.e. the output of the build process of FIG. 1) is deployed, which is partly automated in prior art. Then, in a further step 220 a test case list is created manually. Next, the program is executed multiple times using the parameters of each test case comprised of a test case list 235 in a sequence of test runs, see step 230. Further, the results of the test runs are evaluated in a step 240.
In more detail, so-called “code coverage” itself is also a well-known prior art technique in software testing.
Semantic clustering is also a known technique, see for example: (http://www.google.de/url?sa=t&source=web&ct=res&cd=3&ved=0CBcQFjAC&url=http %3A %2F%2Fscg.unibe.ch%2Farchive%2Fdrafts%2FKuhn06bSemanticClustering.pdf&ei=-xzFStilOsjFsga33eTZDg&rct=j&q=semantic+clustering&usg=AFQjCNEONw34mOwdmrm6k2 ufA2Nc0PNPLQ)
After the preceding introduction into prior art, the disadvantage of prior art will be described in relation to the present invention as follows next below:
As the use of computer software in performing daily tasks rapidly increases, assessing software reliability through software testing has become an imperative stage in the software development cycle. As is well known, software testing is directed toward eliminating defects (i.e., bugs) in computer software, which if undetected, can create significant negative results.
Typically, software testing process involves testing of each specification document by one or multiple test suites. For simplifying the testing operation and ease of reference, test suites are divided into sections, such as tests, with each test being composed of several test cases. For the same reasons, the specification documents are also divided into, among others, chapters, sections, subsections, and assertions. As used herein, assertions are defined as Boolean expressions designed to convey a necessary behavior of the software program and are typically included in the text. Assertions may be identified by implementation specific terminology such as “must,” “should,” “always,” “optional,” etc.
As the primary goal in testing software application testing is to ensure that the specifications are thoroughly tested, it is crucial to create test suites that when run accumulatively, test the entire specification. Thus far, however, the current state of software testing is incapable of providing a test developer information confirming that a given assertion in the specification document has been tested.
In accordance with the current testing process, the specification document is reviewed by the test group architecture in an attempt to locate all the assertions. Unfortunately, the reviewing process must be performed manually, making this task very tedious and time-consuming. To make the matter worse, merely locating each of the assertions in the specification does not provide the test developer with sufficient information to determine whether a particular assertion has been tested.
Consequently, each assertion must be mapped to the test case designed to test that particular assertion, in an attempt to configure that each and every assertion is tested. Unfortunately, this mapping operation is also very lengthy and tedious as it is performed manually by the test group architecture. Furthermore, despite being time consuming and a waste in resources, the mapping operation provides the test developer very limited information about the test coverage of a specification.
U.S. Pat. No. 7,210,066 B2 addresses these problems and discloses a computer method for software testing using test coverage data focussing to ensure that a specification of an software application (a static document, created upfront the implementation process) is optimally tested by a test suite (a collection of test cases). This prior art method therefore matches static information derived of the specification (the assertions in the assertion document) with a static file defining a relation what assertions are tested by which test case.
However, in that published document the code coverage information is generated manually by creating so-called assertions from a (hand written) specification document. So when the specification is changed, the assertions have to be changed as well. Assertions are assigned to test cases, so that one has the relation between certain parts in the specification and test cases that test this specific part.
Disadvantageously, these processes of building a program and testing the program are not connected today and do not share information. So, the before-mentioned work is quite tedious and error-prone.